Discrete Time Markov Chains
Markov processes
We have seen that sequences are easiest discrete time random processes. However, they don’t capture correlation well.
Definition 1. A stochastic process \(X:\Omega\to\sX^T\) with state space \(\sX\) and ordered index set \(T\) is Markov if conditioned on the present state \(X_t\), future \(\sigma(X_u, u > t)\) is independent of the past \(\sigma(X_s, s< t)\). We denote the history of the process until time \(t\) as \(\sF_t \triangleq \sigma(X_s, s \le t)\). That is, for any Borel measurable set \(B \in \sB(\sX)\) any two indices \(u > t\), we have
Remark 1. We next re-write the Markov property more explicitly for the process \(X:\Omega\to\R^\R\). For all \(x, y \in \sX\), finite set \(S \subseteq \R\) such that \(\max S < t < u\), and \(H_S(x_S) \triangleq \cap_{s \in S}\set{X_s \le x_s} \in \sF_t\), we have
Discrete time Markov chains
Definition 2. For a state space \(\sX \subseteq \R\) and the random sequence \(X: \Omega \to \sX^{\Z_+}\), we define the history until time \(n \in \Z_+\) as \(\sF_n \triangleq \sigma(X_1, \dots, X_n)\).
Remark 2. Recall that the event space \(\sF_n\) is generated by the historical events \(A_X(x) \triangleq \cap_{i=0}^n\set{X_i \le x_i}\) where \(x \in \R^{n+1}\).
Remark 3. When the state space \(\sX\) is countable, the event space \(\sF_n\) is generated by the historical events \(H_n(x) \triangleq \cap_{i=0}^n\set{X_i = x_i}\), where \(x \in \sX^{n+1}\). That is, \(\sF_n = \sigma(H_n(x):x \in \sX^{n+1})\)
Definition 3. For a countable set \(\sX\), a discrete-valued random sequence \(X: \Omega \to \sX^{\Z_+}\) is called a discrete time Markov chain (DTMC) if for all positive integers \(n \in \Z_+\), all states \(x,y \in \sX\), and any historical event \(H_{n-1} = \cap_{m=0}^{n-1}\set{X_m = x_m} \in \sF_n\) for \((x_0, \dots, x_{n-1}) \in \sX^{n}\), the process \(X\) satisfies the Markov property
Remark 4. The above definition is equivalent to \(P(\set{X_{n+1} \le x}\given \sF_n) = P(\set{X_{n+1} \le x}\given \sigma(X_n))\) for discrete time discrete state space Markov chain \(X\), since \(\sF_n = \sigma(H_n(x): x \in \sX^n)\) and \(\sigma(X_n) = \sigma(\set{X_n = x}, x \in \sX)\).
Example 4 (Random Walk). A random walk \(S:\Omega\to\sX^\N\) with independent step-size sequence \(X:\Omega\to\sX^\N\), is Markov for a countable state space \(\sX\) that is closed under addition. Given a historical event \(H_{n-1}(s) \triangleq \cap_{k=1}^{n-1}\set{S_k = s_k}\) and the current state \(\set{S_n = s_n}\), we can write the conditional probability The equality in the second line follows from the independence of the step-size sequence. In particular, from the independence of \(X_{n+1}\) from the collection \(\sigma(S_0, X_1, \dots, X_n) = \sigma(S_0, S_1, \dots, S_n)\).
Transition probability matrix
Definition 5. We denote the set of all probability mass functions over a countable state space \(\sX\) by \(\cM(\sX)\triangleq \set{\nu \in [0,1]^\sX: \sum_{x\in \sX}\nu_x = 1}.\)
Definition 6. The transition probability matrix at time \(n\) is denoted by \(P(n) \in [0,1]^{\sX \times \sX}\), such that its \((x,y)\)th entry is denoted by \(p_{xy}(n) \triangleq P(\set{X_{n+1} = y}\given\set{X_n = x})\), that is the transition probability of a discrete time Markov chain \(X\) from a state \(x \in \sX\) at time \(n\) to state \(y \in \sX\) at time \(n+1\).
Remark 5. We observe that each row \(P_{x}(n) \triangleq (p_{xy}(n): y \in \sX) \in\cM(\sX)\) is the conditional distribution of \(X_{n+1}\) given the event \(\set{X_n = x}\).
Definition 7. A matrix \(A \in \R_+^{\sX \times \sX}\) with non-negative entries is called sub-stochastic if the row-sum \(\sum_{y \in \sX} a_{xy} \le 1\) for all rows \(x \in \sX\). If the above property holds with equality for all rows, then it is called a stochastic matrix. If matrices \(A\) and \(A^T\) are both stochastic, then the matrix \(A\) is called doubly stochastic.
Remark 6. We make the following observations for the stochastic matrices.
Every probability transition matrix \(P(n)\) is a stochastic matrix.
All the entries of a sub-stochastic matrix lie in \([0,1]\).
Each row \(A_x \triangleq (a_{xy}: y \in \sX)\) of the stochastic matrix \(A \in \R_+^{\sX \times \sX}\) belongs to \(\cM(\sX)\).
Every finite stochastic matrix has a right eigenvector with unit eigenvalue. This can be observed by taking \(\bU^T = \begin{bmatrix}1 & \dots & 1\end{bmatrix}\) to be an all-one vector of length \(\abs{\sX}\). Then we see that \(A \bU = \bU\), since
Every finite doubly stochastic matrix has a left and right eigenvector with unit eigenvalue. This follows from the fact that finite stochastic matrices \(A\) and \(A^T\) have a common right eigenvector \(\bU\). It follows that \(A\) has a left eigenvector \(\bU^T\).
For a probability transition matrix \(P(n)\), we have \(\sum_{y \in \sX}f(y)p_{xy}(n) = \E[f(X_{n+1})\mid\set{X_n =x}].\)
Homogeneous Markov chains
In general, not much can be said about Markov chains with index dependent transition probabilities. We consider the simpler case where the transition probabilities \(p_{xy}(n)= p_{xy}\) are independent of the index.
Definition 8. A discrete time Markov chain with the probability transition matrix \(P(n)\) that is independent of the index, is called time homogeneous.
Example 9 (Integer random walk). For a one-dimensional integer valued random walk \(S: \Omega \to \Z^\N\) with unit step size sequence \(X: \Omega \to \set{-1,1}^\N\) such that \(P\set{X_1 = 1} = p\), the transition operator \(P \in [0,1]^{\Z\times\Z}\) is given by the entries \(p_{xy} = p\indicator{y=x+1}+(1-p)\indicator{y=x-1}\) for all \(x,y \in\Z\).
Example 10 (Sequence of experiments). Consider a random sequence of experiment outcomes \(X: \Omega\to\set{0,1}^{\Z_+}\), such that \(P(\set{X_{n+1} = 0}\mid\set{X_n=0})= 1-q\) and \(P(\set{X_{n+1} = 1}\mid\set{X_n=1})= 1-p\) for all \(n \in \Z_+\). Then, we can write the probability transition matrix as
Definition 11. Consider a time homogeneous Markov chain \(X: \Omega \to \sX^{\Z_+}\) with countable state space \(\sX\) and transition matrix \(P\). We would respectively denote the conditional probability of events and conditional expectation of random variables, conditioned on the initial state \(\set{X_0 = x}\), by
Proposition 12. Conditioned on the initial state, any finite dimensional distribution of a homogeneous Markov chain is stationary. That is, for any finite \(n,m \in \Z_+\) and states \(x_0, \dots, x_n \in \sX\), we have
Proof. Proof. Consider a homogeneous Markov chain \(X:\Omega\to\sX^{\Z_+}\) with natural filtration \(\sF_\bullet\) such that \(\sF_n \triangleq \sigma(X_0, \dots, X_n)\) for each \(n\in\N\). Using the property of conditional probabilities and Markovity of \(X\), we can write the conditional probability of sample path \((X_1, \dots, X_n)\) given the event \(\set{X_0 = x_0}\) as Using the property of conditional probabilities and Markovity of \(X\), we can write the conditional probability of sample path \((X_{m+1}, \dots, X_{m+n})\) given the event \(\set{X_m = x_0}\) as From time-homogeneity of transition probabilities of Markov chain \(X\), it follows that both the transition probabilities are identical and equal to \(\prod_{i=1}^np_{x_{i-1},x_i}\). ◻
Corollary 13. The \(n\)-step transition probabilities are stationary for any homogeneous Markov chain. That is, for any states \(x_0,x_n \in \sX\) and \(n,m \in \N\), we have \(P(\set{X_{n+m} = x_n}\mid\set{ X_m = x_0}) = P(\set{X_{n} = x_n}\mid\set{ X_0 = x_0}).\)
Proof. Proof. It follows from summing over all possible paths \((X_0, \dots, X_n)\) and \((X_m, \dots, X_{m+n})\). In particular, we can partition events \(\set{X_n=x_n}\) and \(\set{X_{m+n}=x_n}\) in terms of unions over disjoint paths The result follows from the countable additivity of conditional probability for disjoint events of taking distinct paths, and the fact that probability of taking same path is identical for both sums. ◻
Transition graph
A time homogeneous Markov chain \(X: \Omega\to\sX^\N\) with a probability transition matrix \(P\), is sometimes represented by a directed weighted graph \(G = (\sX, E, w)\), where the set of nodes in the graph \(G\) is the state space \(\sX\), and the set of directed edges is the set of possible one-step transitions indicated by the initial and the final state, as In addition, this graph has a weight \(w_e = p_{xy}\) on each edge \(e = [x,y\rangle \in E\).
Example 14 (Integer random walk). The time homogeneous Markov chain in Example [exmp:IRW] can be represented by an infinite state weighted graph \(G = (\Z, E, w)\), where the edge set is We have plotted the sub-graph of the entire transition graph for states \(\set{-1,0,1}\) in Figure 1.
Example 15 (Sequence of experiments). The time homogeneous Markov chain in Example [exmp:MarkovBerExp] can be represented by the following two-state weighted transition graph \(G = (\set{0,1}, E, w)\), plotted in Figure 2.
Random mapping theorem
We saw some example of Markov processes where \(X_n = X_{n-1} + Z_n\), and \(Z:\Omega\to\Lambda^\N\) is an sequence, independent of the initial state \(X_0\). We will show that any discrete time Markov chain is of this form, where the sum is replaced by arbitrary functions.
Theorem 16 (Random mapping theorem). For any DTMC \(X:\Omega\to\sX^{\Z_+}\), there exists an sequence \(Z \in \Lambda^\N\) and functions \(f_n: \sX \times \Lambda \to \sX\) such that \(X_{n} = f_n(X_{n-1}, Z_{n})\) for all \(n \in \N\).
Remark 7. A random mapping representation of a transition matrix \(P(n)\) on state space \(\sX\) is a function \(f_n: \sX \times \Lambda \to \sX\), along with a random variable \(Z_n:\Omega\to\Lambda\), satisfying for all \(x,y \in \sX\),
Proof. Proof. It suffices to show that every transition matrix \(P(n)\) has a random mapping representation. Then, for the mapping \(f_n\) and the sequence \(Z:\Omega\to\Lambda^\N\), we would have \(X_n = f_n(X_{n-1}, Z_n)\) for all \(n \in \N\).
Let \(\Lambda \triangleq [0,1]\), and we choose the uniform sequence \(Z:\Omega\to\Lambda^\N\). Since \(\sX\) is countable, it can be ordered. We let \(\sX = \N\) without any loss of generality. We set \(F_{xy}(n) \triangleq \sum_{w \le y}p_{xw}(n)\) and define function \(f_n:\sX\times\Lambda\to\sX\) for all pairs \((x,z)\in\sX\times\Lambda\) by Since \(f_n(x,Z_n)\) is a discrete random variable taking value \(y \in \sX\), iff the uniform random variable \(Z_n\) lies in the interval \((F_{x, y-1}(n), F_{x,y}(n)]\). That is, the event \(\set{f_n(x,Z_n) = y} = \set{Z_n \in (F_{x, y-1}(n), F_{x,y}(n)]}\) for all \(y \in \sX\). It follows that ◻